Papers with automatic evaluation method
uBLEU: Uncertainty-Aware Automatic Evaluation Method for Open-Domain Dialogue Systems (2020.acl-srw)
Copied to clipboard
| Challenge: | Existing evaluation metrics for text generation tasks do not consider uncertain responses without writing additional reference responses by hand. |
| Approach: | They propose a human-aided, uncertainty-aware evaluation method for open-domain dialogue systems, BLEU. |
| Outcome: | The proposed method is comparable to existing methods on Twitter and improves state-of-the-art evaluation method RUBER. |
Does GPT-3 Generate Empathetic Dialogues? A Novel In-Context Example Selection Method and Automatic Evaluation Metric for Empathetic Dialogue Generation (2022.coling-1)
Copied to clipboard
| Challenge: | Empathy is a multi-dimensional concept consisting of cognitive and affective aspects. |
| Approach: | They propose two new in-context example selection methods that utilize emotion and situational information. |
| Outcome: | The proposed method is effective in measuring the degree of human empathy. |
Near-Negative Distinction: Giving a Second Life to Human Evaluation Datasets (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for evaluating progress in natural language generation tasks are expensive, difficult to reproduce, and non-reusable. |
| Approach: | They propose a new automatic evaluation method for NLG called Near-Negative Distinction that repurposes prior human annotations into NND tests. |
| Outcome: | The proposed method achieves higher correlation with human judgments than standard NLG evaluation metrics. |
Automatic Machine Translation Evaluation using Source Language Inputs and Cross-lingual Language Model (2020.acl-main)
Copied to clipboard
| Challenge: | Existing methods for machine translation evaluation use source sentences as pseudo references instead of word symbols. |
| Approach: | They propose an automatic machine translation evaluation method that uses source sentences as pseudo references instead of source sentences. |
| Outcome: | The proposed method achieves higher correlation with human judgments than baseline evaluation method that uses only hypothesis and reference sentences. |
ArxivDIGESTables: Synthesizing Scientific Literature into Tables using Language Models (2024.emnlp-main)
Copied to clipboard
Benjamin Newman, Yoonjoo Lee, Aakanksha Naik, Pao Siangliulue, Raymond Fok, Juho Kim, Daniel Weld, Joseph Chee Chang, Kyle Lo
| Challenge: | Using language models (LMs) can generate literature review tables by decomposing it into separate schema and value generation steps. |
| Approach: | They propose a framework that leverages language models to perform literature review table generation by decomposing it into separate schema and value generation steps. |
| Outcome: | The proposed framework decomposes the task into two sub-tasks: schema generation and value generation. |
MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMs (2025.findings-acl)
Copied to clipboard
Kaustubh Deshpande, Ved Sirdeshmukh, Johannes Baptist Mols, Lifeng Jin, Ed-Yeremai Hernandez-Cardona, Dean Lee, Jeremy Kritz, Willow E. Primack, Summer Yue, Chen Xing
| Challenge: | Existing evaluation frameworks for large language models have limited coverage for multi-turn conversations . multi-turned conversations require accurate instruction following, context allocation, and in-context reasoning at the same time. |
| Approach: | They propose a benchmark to evaluate large language models' ability to conduct multi-turn conversations with humans. |
| Outcome: | The proposed benchmarks achieve near perfect scores on existing benchmarks but only a 41.4% accuracy on the frontier models. |